-Divergences and Related Distances

Derivation of tight bounds on f -divergences and related distances is of interest in information theory and statistics. This paper improves some existing bounds on f -divergences. In some cases, an alternative approach leads to a simplified proof of an existing bound. Following bounds on the chi-squared divergence, an improved version of a reversed Pinsker’s inequality is derived for an arbitrary pair of probability distributions on a finite set. Following bounds on the relative entropy and Jeffreys’ divergence, a tightened inequality for lossless source coding is derived and considered. Finally, a new inequality relating f -divergences is derived and studied. Index Terms – Bhattacharyya distance, Chernoff information, chi-squared divergence, f -divergence, Hellinger distance, Jeffreys’ divergence, lossless source coding, relative entropy (Kullback-Leibler divergence), total variation distance.